Tutorial for cross-compilation #160

domenkozar · 2021-06-14T16:56:13Z

cloudflare-workers-and-pages · 2021-06-14T16:56:57Z

Deploying with Cloudflare Pages

Latest commit:	`608df83`
Status:	✅ Deploy successful!
Preview URL:	https://5031d9ad.nix-dev.pages.dev

View logs

sternenseemann

Rough first pass.

sternenseemann · 2021-06-14T17:02:24Z

source/tutorials/cross-compilation.rst

+       `autoconf terminology <https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Hosts-and-Cross_002dCompilation.html>`_.
+
+.. note:: macOS/Darwin is a special case, as not the whole OS is Open Source. 
+          It's only possible to cross-compile between ``aarch64-darwin`` and ``x86_64-darwin``.


Technically only from x86_64-darwin to aarch64-darwin. Not sure if I'd mention this here though, as aarch64-darwin is far from mature at this point and cross compilation is not working at all in any channel currently…

Technically only from x86_64-darwin to aarch64-darwin.

Is that because we're missing pkgsCross.x86_64-darwin or something else?

Not sure if I'd mention this here though, as aarch64-darwin is far from mature at this point and cross compilation is not working at all in any channel currently…

Since the fix from @alyssais should work, the whole cross-compilation is not well supported but I'd like this tutorial to show that it's entirely feasible with enough maintainers. WDYT?

Is that because we're missing pkgsCross.x86_64-darwin or something else?

I'm not sure as to why exactly, but I can confirm that nixpkgs can't even cross compile from x86_64-darwin to x86_64-darwin due to an evaluation problem very early in stdenv.

the whole cross-compilation is not well supported but I'd like this tutorial to show that it's entirely feasible with enough maintainers

Still, aarch64-linux for example has had more time to mature, accumulate fixes etc. Maybe not mentioning aarch64-darwin saves someone a bit of frustration?

How about a disclaimer that we've just added support for it?

I've added a disclaimer and move this further down.

source/tutorials/cross-compilation.rst

sternenseemann · 2021-06-14T17:05:55Z

source/tutorials/cross-compilation.rst

+as you'd then like to build a compiler on the build platform, compile code on the
+target plaform and run the final executable on the host platform.
+
+Since that's rarely needed, we'll treat target platform the same as the build.


Just write that you ignore the platform for this tutorial. The default case of targetPlatform is the same as hostPlatform afaik.

Also interesting to note: On the outermost interface nixpkgs doesn't support targetPlatform, it only accepts localSystem (~ build) and crossSystem (~ host) since cross compiling a cross compiler is a rare use case outside of stdenv bootstrapping.

I'd like to clarify since it took me quite some time to figure out why target is not important and explain why it can be ignored to save confusion. The user will eventually stumble upon it as they dive into cross-compilation.

I've thought about this yesterday and maybe it would actually help the tutorial to explain target completely. This would make it very easy to clear the following things up:

Why are binutils, pkg-config and cc prefixed? What does the prefix mean? This is quite easy to answer if you explain the relation of the target prefix to the target platform

What is buildPackages? Here having an understanding of target is nice: buildPackages usually has build and host platform equivalent to localSystem, but its target is crossSystem — having an understanding of target automatically explains how we bootstrap our cross package sets

Agreed! I don't think I know enough to really explain those, unfortunately.

sternenseemann · 2021-06-14T17:06:14Z

source/tutorials/cross-compilation.rst

+-----------------------------
+
+The build platform is determined automatically by Nix
+as it can just guess it during the configure phase.


You can also set it by passing localSystem.

In what cases is that really needed? In most cases guessing should be sufficient?

The two main use cases would be remote build (if you have a builder with a different platform than your local machine) and testing evaluation of cross-related things by simulating being on another platform.

I'd like to mention that then in #157 tutorial (and consider developing nixpkgs out of scope here).

source/tutorials/cross-compilation.rst

domenkozar · 2021-06-15T10:29:53Z

cc @zimbatm @zupo (for all the previous help with tutorials!) @Ericson2314 @matthewbauer @alyssais (for cross-compilation expertise) @otobrglez @boxofrox (for thumbs up on the tutorial issue)

k-bx · 2021-06-15T10:49:38Z

Apologies for a noob question, but I'm getting this:

$ nix-shell --run '$CC hello.c -o hello' cross-compile-shell.nix
error: undefined variable 'file' at /home/kb/workspace/nix-crosscompile-test/cross-compile-shell.nix:8:36

$ cat cross-compile-shell.nix
{ nixpkgs ? fetchTarball "https://github.com/NixOS/nixpkgs/archive/bba3474a5798b5a3a87e10102d1a55f19ec3fca5.tar.gz"
, pkgs ? (import nixpkgs {}).pkgsCross.aarch64-multiplatform
}:

# pkgs.callPackage is needed due to https://github.com/NixOS/nixpkgs/pull/126844
pkgs.callPackage ({ mkShell, zlib, pkg-config }: mkShell {
  # these tools run on the build platform, but are configure to target the target platform
  nativeBuildInputs = [ pkg-config file ];
  # libraries needed for the target platform
  buildInputs = [ zlib ];
}) {}

Am I doing something wrong?

nh2 · 2021-06-15T10:51:12Z

@k-bx In

nativeBuildInputs = [ pkg-config file ];

the pkg-config comes from the lambda arguments { mkShell, zlib, pkg-config }: mkShell, but file doesn't. You need to add it to those arguments so that it is in scope.

source/tutorials/cross-compilation.rst

domenkozar · 2021-06-15T10:52:26Z

Apologies for a noob question, but I'm getting this:

$ nix-shell --run '$CC hello.c -o hello' cross-compile-shell.nix
error: undefined variable 'file' at /home/kb/workspace/nix-crosscompile-test/cross-compile-shell.nix:8:36

$ cat cross-compile-shell.nix
{ nixpkgs ? fetchTarball "https://github.com/NixOS/nixpkgs/archive/bba3474a5798b5a3a87e10102d1a55f19ec3fca5.tar.gz"
, pkgs ? (import nixpkgs {}).pkgsCross.aarch64-multiplatform
}:

# pkgs.callPackage is needed due to https://github.com/NixOS/nixpkgs/pull/126844
pkgs.callPackage ({ mkShell, zlib, pkg-config }: mkShell {
  # these tools run on the build platform, but are configure to target the target platform
  nativeBuildInputs = [ pkg-config file ];
  # libraries needed for the target platform
  buildInputs = [ zlib ];
}) {}

Am I doing something wrong?

Not at all, I added file and didn't do the full test again. Should be fixed now!

k-bx · 2021-06-15T10:55:45Z

Aha! It worked now, thanks. Sorry for hijacking the thread, but I'm trying to build hello.c for my Raspberry Pi Zero. What should I put instead of aarch64-multiplatform in that cross-compile-shell.nix exactly?

UPDATE: ok, figured it's just raspberryPi hehe :)

domenkozar · 2021-06-15T11:09:27Z

Aha! It worked now, thanks. Sorry for hijacking the thread, but I'm trying to build hello.c for my Raspberry Pi Zero. What should I put instead of aarch64-multiplatform in that cross-compile-shell.nix exactly?

UPDATE: ok, figured it's just raspberryPi hehe :)

https://b5839745.nix-dev.pages.dev/tutorials/cross-compilation#determining-the-host-platform is supposed to teach how to determine platform config.

https://b5839745.nix-dev.pages.dev/tutorials/cross-compilation#choosing-the-host-platform-with-nix is supposed to teach how to pick the right attribute with that platform config in mind.

Let me know where my explaining went south :)

k-bx · 2021-06-15T11:48:55Z

@domenkozar oh, I see. I was confused by the terminology, assuming that the host is the machine that does the building, and the "target" (or some other term) is my raspberryPi, since that command runs nix to determine the architecture, and I would assume you wouldn't need to install Nix on raspberry pi. Now I see that you would.

k-bx · 2021-06-15T12:08:45Z

Again apologies for spamming, just trying to get that raspberry pi zero hello world to work.

Installing nix is not an option (sorry, there is no binary distribution of Nix for your platform) to determine the host platform. I would assume it's armv6*. If I were to try armv6l-unknown-linux-gnueabihf mentioned in that tutorial, I'd get an error of that attribute missing:

$ nix-shell --run '$CC hello.c -o hello' cross-compile-shell.nix
error: attribute 'armv6l-unknown-linux-gnueabihf' missing, at /home/kb/workspace/nix-crosscompile-test/cross-compile-shell.nix:2:10
(use '--show-trace' to show detailed location information)

$ cat ./cross-compile-shell.nix
{ nixpkgs ? fetchTarball "https://github.com/NixOS/nixpkgs/archive/bba3474a5798b5a3a87e10102d1a55f19ec3fca5.tar.gz"
, pkgs ? (import nixpkgs {}).pkgsCross.armv6l-unknown-linux-gnueabihf
}:

# pkgs.callPackage is needed due to https://github.com/NixOS/nixpkgs/pull/126844
pkgs.callPackage ({ mkShell, zlib, pkg-config, file }: mkShell {
  # these tools run on the build platform, but are configure to target the target platform
  nativeBuildInputs = [ pkg-config file ];
  # libraries needed for the target platform
  buildInputs = [ zlib ];
}) {}

Am I not reading the tutorial correctly? Again, sorry for noob stuff.

domenkozar · 2021-06-15T13:53:45Z

@k-bx thanks, this is really useful! I've added a sentence to hint how to get platform config if it's not possible to use Nix:

If you can't install Nix, find a way to run config.guess (usually comes with
the autoconf package) from the OS you're able to run on the host platform.

pkgsCross attributes are made-up names that map into the platform config. I've clarified that in the tutorial.

Unfortunately, there's no simple way to show us all attribute name -> platform config pairs.

language edit

k-bx · 2021-06-15T14:52:37Z

Aha! Interestingly, config.guess is not in PATH, but found it in /usr/share/misc/config.guess. It's armv6l-unknown-linux-gnueabihf indeed. Due to https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix#L38 the value raspberryPi was indeed the right one. However, if I copy the resulting executable on my rpi0, I get this:

pi@raspberrypi:~ $ ls -la
total 88
drwxr-xr-x 10 pi   pi    4096 Jun 15 12:45 .
drwxr-xr-x  3 root root  4096 Mar  4 22:47 ..
-rwxr-xr-x  1 pi   pi   11600 Jun 15 15:46 hello
pi@raspberrypi:~ $ ./hello 
-bash: ./hello: No such file or directory
pi@raspberrypi:~ $ ldd ./hello 
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so (0xb6f86000)
        libc.so.6 => /usr/lib/arm-linux-gnueabihf/libc.so.6 (0xb6e38000)
        /nix/store/bs0yybprj2jdz4gk4g2738pnfbraaxh2-glibc-2.31-74-armv6l-unknown-linux-gnueabihf/lib/ld-linux-armhf.so.3 => /lib/ld-linux-armhf.so.3 (0xb6f99000)

That's why I thought raspberryPi wasn't the right value initially. Do you know what could be the reason for this?

Unfortunately, there's no simple way to show us all attribute name -> platform config pairs.

That is indeed one of the pain-points of Nix: you see the final spell, but it's hard to inspect what other keys and values are available. I hope to dive into Nix near time and see if I can help improve this situation.

sternenseemann · 2021-06-15T15:13:58Z

@k-bx The no such file or directory is the dynamic linker telling you that it is not finding the libraries linked against. To fix that you can either a) compile statically by passing -static to $CC or b) copy all dependencies of your executable. Finding this out for a binary built with nix-shell is a bit annoying, but if you have a proper derivation you could use nix-store --requisites <realised store path>.

The topic of dynamic linking and dependencies should be mentioned in the tutorial as well. For the example it is probably easier just using static linking though.

k-bx · 2021-06-15T15:34:05Z

@sternenseemann thank you! May I just mention that adding -static will produce an error:

/nix/store/s8j8w9kik99ffxcgwz3bdpk71ijr3qj7-armv6l-unknown-linux-gnueabihf-binutils-2.31.1/bin/armv6l-unknown-linux-gnueabihf-ld: cannot find -lc

I've tried adding glibc, that didn't help. libc is not found when accepted as lambda argument, not sure what exactly to do next (again, probably due to lack of linker knowledge/experience).

sternenseemann · 2021-06-15T15:58:31Z

@k-bx Oh yeah, you need to use the static output for glibc, i. e. glibc.static or alternatively musl, but in that case you probably want to switch stdenv (static compilation is also a lot easier if you actually use a static stdenv of course…).

source/tutorials/cross-compilation.rst

k-bx · 2021-06-15T16:51:25Z

@sternenseemann thank you! Changing buildInputs argument to glibc.static indeed helped!

I guess I finally have a starting point of how to get from here to a point where I can cross-compile my Rust project that is using some native libraries :)

sternenseemann · 2021-06-15T17:09:55Z

@k-bx Yeah, this is something we should take into consideration as well. There is some special casing to be done for rust possibly, I'm not quite sure.

In general the way would be:

Package your project normally as a nix derivation, take extra care to properly distinguish between nativeBuildInputs and buildInputs (see manual) and add it via an overlay to nixpkgs
Then compiling pkgsCross.<platform>.my-package should just work (if everything is supported)
- For shared executables (the default case) you need to copy the entire closure (i. e. what nix-store --requisites prints) while preserving its location (i. e. everything must be under /nix/store). If you have nix on the other machine, you can use nix-copy-closure for this.
- static executables can just be copied as is, you can build your derivation statically using pkgsCross.<platform>.pkgsStatic.my-package

source/tutorials/cross-compilation.rst

Co-authored-by: John Ericson <git@JohnEricson.me>

domenkozar · 2021-06-16T16:08:18Z

Thanks everyone for the feedback!

I've included most of it in the last commit, please take another look!

source/tutorials/cross-compilation.rst

domenkozar · 2021-06-17T08:58:55Z

@sternenseemann thank you! Changing buildInputs argument to glibc.static indeed helped!

I guess I finally have a starting point of how to get from here to a point where I can cross-compile my Rust project that is using some native libraries :)

The example for developer environment uses now a cross compiler that emits static binaries. Thanks for bringing this up!

domenkozar · 2021-06-17T09:02:32Z

I'm going to merge this in a few hours to prevent perfect becoming the enemy of the good.

Let me know if I missed some feedback and happy to address it further, but I might convert it into issues to keep things going.

domenkozar force-pushed the cross-compilation-tutorial branch from c9bb3fd to 068416c Compare June 14, 2021 16:56

Tutorial for cross-compilation

365c398

domenkozar force-pushed the cross-compilation-tutorial branch from 068416c to 365c398 Compare June 14, 2021 17:00

sternenseemann suggested changes Jun 14, 2021

View reviewed changes

domenkozar mentioned this pull request Jun 15, 2021

Fully static Haskell executables - overview issue NixOS/nixpkgs#43795

Open

60 tasks

sternenseemann reviewed Jun 15, 2021

View reviewed changes

source/tutorials/cross-compilation.rst Outdated Show resolved Hide resolved

feedback

a928ff1

domenkozar force-pushed the cross-compilation-tutorial branch from f9922e9 to a928ff1 Compare June 15, 2021 10:51

Language edit

df7523b

domenkozar force-pushed the cross-compilation-tutorial branch from a928ff1 to d8ca836 Compare June 15, 2021 13:49

feedback

4604ab5

domenkozar force-pushed the cross-compilation-tutorial branch from d8ca836 to 4604ab5 Compare June 15, 2021 13:53

domenkozar mentioned this pull request Jun 15, 2021

Cross-compilation package sets should be named based on the platform config NixOS/nixpkgs#126949

Open

domenkozar added 2 commits June 15, 2021 16:47

Merge branch 'cross-compilation-tutorial' into patch-2

9d4e7e6

Merge pull request #161 from tzarsquared/patch-2

46fd5b5

language edit

Ericson2314 reviewed Jun 15, 2021

View reviewed changes

source/tutorials/cross-compilation.rst Outdated Show resolved Hide resolved

Merge branch 'master' into cross-compilation-tutorial

7b7c836

pingiun reviewed Jun 16, 2021

View reviewed changes

source/tutorials/cross-compilation.rst Outdated Show resolved Hide resolved

source/tutorials/cross-compilation.rst Outdated Show resolved Hide resolved

Update source/tutorials/cross-compilation.rst

3729167

Co-authored-by: John Ericson <git@JohnEricson.me>

Ericson2314 reviewed Jun 16, 2021

View reviewed changes

source/tutorials/cross-compilation.rst Outdated Show resolved Hide resolved

domenkozar force-pushed the cross-compilation-tutorial branch 2 times, most recently from c4e37e2 to 665302f Compare June 17, 2021 08:57

feedback

608df83

domenkozar force-pushed the cross-compilation-tutorial branch from 665302f to 608df83 Compare June 17, 2021 09:01

domenkozar merged commit ec9298d into master Jun 17, 2021

github-actions bot locked and limited conversation to collaborators Jun 17, 2021

fricklerhandwerk deleted the cross-compilation-tutorial branch October 9, 2023 15:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tutorial for cross-compilation #160

Tutorial for cross-compilation #160

domenkozar commented Jun 14, 2021 •

edited

Loading

cloudflare-workers-and-pages bot commented Jun 14, 2021 •

edited

Loading

sternenseemann left a comment

sternenseemann Jun 14, 2021

domenkozar Jun 15, 2021

sternenseemann Jun 15, 2021

domenkozar Jun 16, 2021

domenkozar Jun 17, 2021

sternenseemann Jun 14, 2021

domenkozar Jun 15, 2021

sternenseemann Jun 15, 2021

domenkozar Jun 15, 2021

sternenseemann Jun 14, 2021

domenkozar Jun 15, 2021

sternenseemann Jun 15, 2021

domenkozar Jun 15, 2021

domenkozar commented Jun 15, 2021 •

edited

Loading

k-bx commented Jun 15, 2021

nh2 commented Jun 15, 2021

domenkozar commented Jun 15, 2021

k-bx commented Jun 15, 2021 •

edited

Loading

domenkozar commented Jun 15, 2021

k-bx commented Jun 15, 2021

k-bx commented Jun 15, 2021 •

edited

Loading

domenkozar commented Jun 15, 2021

k-bx commented Jun 15, 2021

sternenseemann commented Jun 15, 2021 •

edited

Loading

k-bx commented Jun 15, 2021 •

edited

Loading

sternenseemann commented Jun 15, 2021

k-bx commented Jun 15, 2021

sternenseemann commented Jun 15, 2021 •

edited

Loading

domenkozar commented Jun 16, 2021

domenkozar commented Jun 17, 2021

domenkozar commented Jun 17, 2021

Tutorial for cross-compilation #160

Tutorial for cross-compilation #160

Conversation

domenkozar commented Jun 14, 2021 • edited Loading

cloudflare-workers-and-pages bot commented Jun 14, 2021 • edited Loading

Deploying with Cloudflare Pages

sternenseemann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenkozar commented Jun 15, 2021 • edited Loading

k-bx commented Jun 15, 2021

nh2 commented Jun 15, 2021

domenkozar commented Jun 15, 2021

k-bx commented Jun 15, 2021 • edited Loading

domenkozar commented Jun 15, 2021

k-bx commented Jun 15, 2021

k-bx commented Jun 15, 2021 • edited Loading

domenkozar commented Jun 15, 2021

k-bx commented Jun 15, 2021

sternenseemann commented Jun 15, 2021 • edited Loading

k-bx commented Jun 15, 2021 • edited Loading

sternenseemann commented Jun 15, 2021

k-bx commented Jun 15, 2021

sternenseemann commented Jun 15, 2021 • edited Loading

domenkozar commented Jun 16, 2021

domenkozar commented Jun 17, 2021

domenkozar commented Jun 17, 2021

domenkozar commented Jun 14, 2021 •

edited

Loading

cloudflare-workers-and-pages bot commented Jun 14, 2021 •

edited

Loading

domenkozar commented Jun 15, 2021 •

edited

Loading

k-bx commented Jun 15, 2021 •

edited

Loading

k-bx commented Jun 15, 2021 •

edited

Loading

sternenseemann commented Jun 15, 2021 •

edited

Loading

k-bx commented Jun 15, 2021 •

edited

Loading

sternenseemann commented Jun 15, 2021 •

edited

Loading